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ABSTRACT 


Program managers throughout the Department of Defense (DoD) are faced with 
technology portfolio management problems. Critical to these efforts is the need to track 
the performance of the technology on a routine, ongoing basis. This thesis focuses on 
solving this general problem in the specific context of the United States Navy’s 
Cryptologic Carry-On Program (CCOP). This study provides a method that can gather 
real world data from United States Naval vessels afloat and use that data to generate 
Return On Investment (ROI) estimates based upon Knowledge Value Added (KVA) 
analysis. This research builds upon the already developed KVA analysis method through 
providing a means by which a constant flow of real world data can feed this process, 
thereby providing an output that is both current and meaningful. The ability of decision 
makers to access this information will provide them with a critical tool that they can 


leverage to help them make wise financial decisions with respect to the CCOP program. 
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I. INTRODUCTION 


A. PURPOSE / PROBLEM STATEMENT 


The objective of this research is to build an implementation plan for collecting, 
retrieving and analyzing data that will be used to perform Knowledge Value Added 
(KVA) analysis on Cryptologic Carry-On Program (CCOP) systems. The output of the 
KVA analysis can be used to generate Return on Investment (ROD) estimates for those 
CCOP systems. The methodology for producing ROI estimates based on KVA analysis 
was developed by Lieutenant Commander Cesar Rios in his thesis, “Return on 
Investment of Information Warfare Systems.” The concept was further refined by 
Lieutenant Hubert Clapp and Lieutenant Ira Lambeth in their thesis, “Using Knowledge 
Value Added (KVA) for Evaluating Cryptologic IT Capabilities: Trial Implementation.” 
This research builds upon the previous work by developing an implementation model that 
will provide a stream of real-world data from U.S. Naval vessels afloat. The importance 
of this research is that a consistent flow of accurate ROI estimates for CCOP systems will 
provide a valuable tool for program managers to gauge the performance of various CCOP 
systems relative to each other and to other types of systems. With this type of knowledge 
available, CCOP acquisition and budget personnel will have a powerful tool that they can 
use to help validate difficult financial decisions. 

Clapp and Lambeth say that KVA “is the underlying foundation used to develop 
and analyze Measures of Performance (MOPs) which are used to quantify and value the 
outputs. A cost and price per unit of output is estimated using the KVA methodology 
which describes all outputs in common units.”’! It is the concept of common units that 
makes this process so powerful because it allows for the comparison of seemingly 


disparate systems through the analysis of identical outputs. 


1 Hubert N. Clapp and Ira D. Lambeth, III, “Using Knowledge Value Added (KVA) for Evaluating 
Cryptologic IT Capabilities: Trial Implementation” (MS thesis, Naval Postgraduate School, 2007). 


1 


Additionally, the market comparable valuation method? is used to estimate 
surrogate revenue pricing to enable an estimate of Return on Investment (ROI) for each 
CCOP system. Previous thesis work has applied this methodology to historical data 
collected from the CCOP systems in use during an 18-month deployment of the USS 
GONZALES (DDG 66). ROI data was analyzed and modeled using GaussSoft™ KVA 
Performance Accounting Modeling Software. 

This research seeks to introduce several possible data-collection procedure 
options that will define and collect the pieces of data required to conduct KVA analysis, 
outline the means by which that data can be retrieved from the operational unit, and 
identify what needs to be accomplished in order to transform that data into usable 
GaussSoft™ input. Consideration will also be given to what portions of the data 
collection and analysis process can and should be automated. Further, this research will 
recommend the option that obtains the required information in the most cost-effective and 
manpower-efficient way while still providing the quality needed to produce reliable and 


accurate output. 


B. BACKGROUND 


This thesis provides an implementation plan for collecting, retrieving and 
analyzing data so that the KVA method can be applied to that data. The KVA method 
was developed and refined in two previous thesis works. The first was developed under 
the direction of Dr. Thomas Housel by LCDR Rios in his thesis titled, “Return on 
Investment Analysis of Information Warfare Systems.’’4 This research was conducted at 
the Naval Postgraduate School (NPS) and focused on developing a KVA analysis method 
that provides ROI estimates. While such a method has applications for any organizational 


process that is technology enabled, the method was specifically applied to the Navy’s 


2 Steven Pratt, Robert Reilly, and Robert Schweihs, Valuing a Business. Fourth Edition. New York: 
McGraw-Hill, 2000. 


3 Hubert N. Clapp and Ira D. Lambeth, III, “Using Knowledge Value Added (KVA) for Evaluating 
Cryptologic IT Capabilities: Trial Implementation” (MS thesis, Naval Postgraduate School, 2007). 


4 Cesar G. Rios, Jr., “Return on Investment Analysis of Information Warfare Systems” (MS thesis, 
Naval Postgraduate School, 2005). 
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CCOP. The KVA method is designed to provide decision makers with ROI estimates 
they can use to evaluate system performance and the value associated with the output 
those systems provide. The second thesis project was developed by LT Ira Lambeth and 
LT Hubert Clapp and was supervised by Dr. Thomas Housel. In their thesis, titled “Using 
Knowledge Value Added (KVA) for Evaluating Cryptologic IT Capabilities: Trial 
Implementation,” they took the KVA method developed by LCDR Rios and applied it to 
a real-world implementation.> They were able to refine the process used to conduct KVA 
analysis on CCOP systems and improve the overall accuracy of the ROI estimates 
produced. 

LCDR Rios’ thesis, “Return on Investment Analysis of Information Warfare 
Systems,” focused on building a foundation for using KVA to analyze performance 
metrics. An abstract from that thesis is below: 

The United States Navy’s Cryptologic Carry-On Program Office manages 

a portfolio of Information Warfare (IW) systems. This research and case 

study demonstrate how the Knowledge Value Added (KVA) Methodology 

can be used to formulate a framework for extracting and analyzing 

performance parameters and measures of effectiveness for each system. 

KVA measures the effectiveness and efficiency of CCOP systems and the 

impact they have on the Intelligence Collection Process (ICP) on board 

U.S. Navy Ships. By analyzing the outputs of the subprocesses involved in 

the ICP in common units of change, a price per unit of output can be 

generated to allocate both cost and revenue at the subprocess level. With 

this level of financial detail, a return on investment (ROI) analysis can be 

conducted for each process, or asset.® 
The second thesis written by LT Clapp and LT Lambeth, “Using Knowledge Value 
Added (KVA) for Evaluating Cryptologic IT Capabilities: Trial Implementation,” is the 
follow-on research into the feasibility of an operational implementation of the above 


concepts. An abstract of it follows: 


This study provides a demonstration of how a software suite that monitors process 
performance and its supporting technology can be implemented to provide 
ongoing return on investment information about CCOP technology. This follow- 


5 Hubert N. Clapp and Ira D. Lambeth, III, “Using Knowledge Value Added (KVA) for Evaluating 
Cryptologic IT Capabilities: Trial Implementation” (MS thesis, Naval Postgraduate School, 2007). 


6 Cesar G. Rios, Jr., “Return on Investment Analysis of Information Warfare Systems” (MS thesis, 
Naval Postgraduate School, 2005). 


on research and trial implementation demonstrate how the Knowledge Value 
Added (KVA) Methodology that is embedded in the performance monitoring 
software is used to formulate a framework for extracting and analyzing 
performance parameters and measures of effectiveness for each CCOP system. 
KVA was used to measure the effectiveness and efficiency of CCOP systems and 
the impact they have on the Intelligence Collection Process (ICP) onboard the 
USS GONZALES.’ 


Due to the high quality of the previous work done on using KVA analysis to 
generate ROI estimates, this thesis is able to focus on how to effectively collect real 
world data on a ship, retrieve that data from the ship and present it for analysis using 
previously developed methods. This introductory chapter serves to highlight areas related 
to the problem, and the background and theoretical frameworks of each. The focus of this 
thesis is in creating procedures for providing a data stream that can then be used by the 


KVA analysis method developed by LCDR Rios, LT Clapp and LT Lambeth. 


1, Intelligence Surveillance and Reconnaissance in the Navy 


The Naval Transformation Roadmap (NTR) of 2003 sets direction for the future 
of Navy Intelligence, Surveillance, and Reconnaissance (ISR). The objective of NTR is 
to completely redesign Intelligence sensor capabilities, operational concepts, processes, 
and organizational relationships and culture’. This redesign of the ISR is to replace 
previous guidance that took little account of an environment in which all branches of the 
military are fully integrated. Escalating costs and the complexity of developing new 
technology dictates that greater coordination and stewardship take place. 

Rising costs combined with shrinking budgets demand that frivolous spending be 
eliminated. Good intelligence saves lives and money so the end result of the NTR is 
projected improvements in Navy ISR capabilities. These improvements will integrate 
Navy ISR with other services in a joint environment to leverage all available resources to 
accomplish the operational mission and fulfill national level strategic objectives. 


Integration will be accomplished by developing systems that are capable of working 


7 Hubert N. Clapp and Ira D. Lambeth, III, “Using Knowledge Value Added (KVA) for Evaluating 
Cryptologic IT Capabilities: Trial Implementation,” (MS thesis, Naval Postgraduate School, 2005) 


8 Department of the Navy. Naval Transformation Roadmap 2003: Assured Access & Power 
Projection...From the Sea. Washington: Dept. of the Navy, 2003, 68-69. 
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across service boundaries, replacing service specific stove-piped models. The Navy’s 
ability to integrate into truly cohesive operations and their ability to field a fully 
integrated ISR program that the joint warfighter can use will continue to be a challenge 
into the future. 

Of course, there are difficulties inherent in changing a business model as large 
and complex as the Navy’s ISR program and technology developers are asked to provide 
new technology that is capable of defeating our enemies, protecting our allies and 
functioning in a networked environment. The cost associated with developing and 
supporting these technologies is substantial. The limit on financial resources makes an 
accurate estimation of a systems capability and worth through scientific means extremely 
valuable. Decision makers need an evaluation method to provide them with some 
measurable output on exactly what is being produced by the technologies in question and 
its value relative to its cost. This is a significant shift away from the pattern of just 
spending millions of dollars on a system that is not well understood. This research uses 
KVA analysis as a way to help with the system valuation problem as it applies to the 


CCOP. 


2. The Cryptologic Carry-on Program 


The Cryptologic Carry-on Program (CCOP) is a product of the Advanced 
Cryptologic Systems Engineering program, which develops state-of-the-art ISR 
capabilities in response to Combatant Command requirements for a quick-reaction 
surface, subsurface and airborne cryptologic carry-on capability. Each CCOP system is a 
complex series of subsystems that often carry classification issues with them. For this 
reason, the CCOP systems are referenced simply by a letter throughout all previous 
research, and will continue to be referenced as such in this research. The design and 
functionality of each CCOPS system was analyzed within a previous research project, 
however, these system specifics are outside of the scope of this paper and were omitted to 


maintain an unclassified level in this analysis. 





9 Cesar G. Rios, Jr., "Return on Investment Analysis of Information Warfare Systems" (MS thesis, 
Naval Postgraduate School, 2005). 


CCOP systems are designed to be flexible and thus have the ability to be installed 
with many different configuration possibilities depending on the platform CCOPS is 
being installed on and its intended usage. LCDR Rios, LT Clapp and LT Lambeth all 
used a standard CCOP load in their KVA method to determine the ROI estimates for 
those CCOP systems. 


3. A Brief Definition of ROI 


ROI analysis is a ratio used for building a financial business case. ROI provides 
decision makers with the ability to evaluate past and future performance of a system or 


organization as illustrated by the following formula.!° 


EARNINGS 


PERCENTAGE ROI = INVESTMENT 


For the above formula, the “earnings” represent the difference between revenue 
and expenses, and “investment” represents the capital and assets of the organizations. The 
ROI then produces a metric to determine how efficiently the capital and assets are 
applied. A high ROI represents a high level of asset allocation towards the business 


objectives. !! 


Clarence Nickerson, a Professor at the Harvard University Graduate School of 
Business Administration, writes, “The value of a business property is dependent on what 
it can produce.”!2 He also states, “in order to judge the value of the wealth created, we 
should take into account the property required to produce it.”!3 These principles have 
long been critical in the business world as the use of ROI is often used to help in the 
determination of how valuable a product or service is relative to its cost. It is logical to 


apply these very same investment principles to the public sectors and military to better 


10 Nickerson, Clarence B. Accounting Handbook for Nonaccountants. 3rd Ed. New York: Van 
Nostrand Reinhold Co., 1986. 632. 


1] Hubert N. Clapp and Ira D. Lambeth, III, “Using Knowledge Value Added (KVA) for Evaluating 
Cryptologic IT Capabilities: Trial Implementation,” (MS thesis, Naval Postgraduate School, 2005) 


12 Nickerson, Clarence B. Accounting Handbook for Nonaccountants. 3rd Ed. New York: Van 
Nostrand Reinhold Co., 1986. 652 


13 bid. 


inform investment decisions. In the previous thesis research conducted by LCDR Rios, 
LT Clapp and LT Lambeth, earnings were defined as the output of the CCOP system 


(reporting), and the investment represents both the system and personnel costs. 


Not all of the systems presently used by Navy ISR are worth the financial the 
human cost required to operate and maintain them. As transformation occurs within Navy 
ISR it provides an opportunity to evaluate the complete range of ISR systems and make 


informed investment decisions based on sound financial principles. 


One of the more complex facets of applying ROI calculation to Navy ISR, and 
CCOP specifically is that the output must be converted into a common unit of analysis. In 
the for profit segment of the private sector financial world this is mostly accomplished 
using dollars, however, intelligence reports don’t convert into dollars therefore some type 
of conversion mechanism is required. To address this issue, an analysis of cost of 
developing business intelligence reports is used to estimate a portion of the “value” of an 
intelligence report. Since various subsystems contained within CCOP have different costs 
associated with building intelligence reports, there are different inherent complexities 
resulting in different human costs to develop the reports. These inherent complexities can 


be handled more effectively by applying the Knowledge Value Added theory. !4 
4. Knowledge Value Added 


Knowledge Value-Added (KVA) theory was created by Dr. Tom Housel (Naval 
Postgraduate School) and Dr. Valery Kanevsky (Agilent Labs). KVA is based on the 
assumption that humans and technology in organizations add value by taking inputs and 


changing them into outputs through core processes. !> 


LT Clapp and LT Lambeth wrote about KVA theory, “KVA is a general theory 
for estimating the value added by knowledge assets, using a methodology that is analytic 
and tautological. It is based on the premise that businesses and other organizations 
produce outputs (e.g., products and services) through a series of processes and 


14 Hubert N. Clapp and Ira D. Lambeth, III, “Using Knowledge Value Added (KVA) for Evaluating 
Cryptologic IT Capabilities: Trial Implementation,” (MS thesis, Naval Postgraduate School, 2007). 


13 T. Housel and A. Bell, Measuring and Managing Knowledge. Boston: McGraw-Hill, 2001, 92-93. 
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subprocesses that change, in some manner, the raw inputs (i.e., labor into services, 
information into reports). KWA explains the changes made on the inputs by 
organizational processes to produce outputs in terms of the equivalent corresponding 
changes in entropy. The concept of entropy is defined in the American Heritage 
Dictionary as a “measure of the degree of disorder [or change] in a closed system.” In the 
business context, it can be used as a surrogate for the amount of changes that a process 


makes to inputs to produce the resulting outputs.’”’!® 
C. RESEARCH OBJECTIVES 


The objective of this research is to develop a data collection method for gathering 
real world CCOP KVA data from any deployed United States Navy ship. This study 
builds upon prior research conducted by LCDR Rios, LT Clapp and LT Lambeth and 
assumes that data will be formatted and analyzed in accordance with the processes as 
described in their study. This research develops an implementation plan for a data 
analysis method developed in the previous studies. The primary goal of this study is to 
provide an implementation process that assists in operationalizing the use of the KVA 
evaluation method in the budgeting process for the United States Navy’s Chief of Naval 
Operations (OPNAV) CCOP Program Office (OPNAV N201) acquisition of information 


warfare systems. 
D. METHODOLOGY 


This thesis provides a data collection method implementation plan that can be 
used to gather real world data from deployed naval units. Previous work in this area 
developed an extremely robust means for using KVA data to determine ROI estimates of 
CCOP systems. However, the previous research lacked the granularity required to begin 
actual collection and analysis of data from the fleet. This work is an attempt to bridge the 
gap between the proposed method and the real world application of that method. The data 


collection implementation plan consists of the following steps: 


16 Hubert N. Clapp and Ira D. Lambeth, III, “Using Knowledge Value Added (KVA) for Evaluating 
Cryptologic IT Capabilities: Trial Implementation,” (MS thesis, Naval Postgraduate School, 2007). 
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. Determine critical pieces of KVA information that need to be captured by 


the ship to conduct an ROI analysis. 


. Analyze data collection assumptions that need to be made in order to 


facilitate shipboard processes concerning the gathering and forwarding of 


data required for conducting KVA analysis. 


Generate three cost effective and manpower efficient options for 


gathering, retrieving, and analyzing the data. 


. Develop a post deployment report, which will produce immediate value 


added to the CCOP program using ROI data. 


THIS PAGE INTENTIONALLY LEFT BLANK 
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Hi. THREE OPTIONS FOR DATA COLLECTION, RETRIEVAL 
AND ANALYSIS 


A. INTRODUCTION 


The thesis conducted by LCDR Rios for Navy CCOP systems was initiated by 
then program officer of United States Navy’s Chief of Naval Operations (CNO) 
Cryptologic Carry-On Program Office (OPNAV N201) LCDR Brian Prevo. LCDR Prevo 
contacted fellow Information Warfare Officer and NPS student, LCDR Cesar Rios, 
concerning a CNO directive to focus on three goals for the following fiscal year: 
Efficiencies, Metrics, and Return on Investment.!7 LCDR Rios’ thesis research under the 
direction of Dr. Thomas Housel constructed the initial framework that facilitated the 
utilization of the KVA method to determine ROI for CCOP systems. LT Clapp and LT 
Lambeth were then able to leverage that foundational work to both baseline and further 
refine their research. LT Clapp and LT Lambeth then tested the feasibility of 
implementing the new model in an operational environment and were able to show, using 
historical data, that such analysis can be completed and will render the desired ROI 
estimates. Based on LT Clapp and LT Lambeth’s work and under the guidance of Dr. 
Thomas Housel, LT Jason Homer constructed three framework options for collecting, 
retrieving and analyzing KVA data from US Navy vessels at sea. The following is a 


synopsis of that research. 
1. Objective 


The overall objective of this thesis work was to develop three viable options for 
collecting, retrieving and analyzing real world CCOP data from US Navy vessels afloat 
with the ultimate goal of feeding a decision support model and methodology to assist in 
the POM/Budgeting process for OPNAV N20’s acquisition of IW CCOP systems. The 
research conducted by LCDR Rios, LT Clapp and LT Lambeth created a method of 





17 Department of the Navy, CCOP Program Briefing. Power Point. Washington: Dept. of the Navy, 
CCOP Program Office (OPNAV N201C), 25 April 2005. 
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producing ROI estimates using KVA analysis that can be used to support decision makers 
by giving them access to important system valuation data. Using their work as a new 
baseline, this thesis work sought to outline an effective process to provide a steady stream 
of CCOP data into the KVA analysis model. Providing this data for analysis will enable 
CCOP acquisition decision makers to use empirical data and KVA analysis to evaluate 


the performance of individual CCOP systems for future investment. 
2. Method 


In previous research conducted by LCDR Rios, LT Clapp and LT Lambeth, 


The Knowledge Value Added (KVA) method was used to develop and 
analyze Measures of Performance (MOPs) which were used to quantify 
and value the outputs of the KVA analysis. A cost-per-output was 
calculated using KVA data in conjunction with market comparable pricing 
to determine a Return on Investment (ROI) for each CCOP sub-system. !8 


This thesis describes three different options for gathering and analyzing real 


world data from US Navy vessels afloat. 
B. RESEARCH QUESTION 


Assuming the research done by LCDR Rios, LT Clapp and LT Lambeth as a 
baseline, how can the data required for KVA analysis of CCOP systems be collected, 
retrieved and analyzed. Additionally, this research will seek to determine how this 


information can be used to benefit both the financial and the operational decision maker. 
C. DETERMINATION OF REQUIRED INFORMATION 


In order to effectively collect, retrieve and analyze real world data there must first 


exist a clear and concise understanding of exactly what information is required to conduct 





18 Hubert N. Clapp and Ira D. Lambeth, III, “Using Knowledge Value Added (KVA) for Evaluating 
Cryptologic IT Capabilities: Trial Implementation,” (MS thesis, Naval Postgraduate School, 2005). 
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KVA analysis. The Klieglight (KL) is a highly classified report that the thesis research 
conducted by LCDR Rios has determined to be an acceptable measurable unit of output 


for any CCOP system. 
i KVA Analysis Process 


The KVA model as applied to CCOP systems is a complex analysis of systems, 
subsystems, and operator involvement, all of which function together to produce a value 
that has been assigned to a common output, namely the KL report. What follows is an 
example of how this process was applied in the research conducted by Lieutenants Clapp 
and Lambeth. This example of the KVA process is provides background information to 
help describe what the KVA analysis produces in terms of evaluation metrics and insight 
into how it functions. For further information, on how KVA analysis generates evaluation 
data see “Using Knowledge Value Added (KVA) for Evaluating Cryptologic IT 
Capabilities: Trial Implementation” by Lieutenants Hubert Clapp and Ira Lambeth. 


A CCOP system consists of one or more subsystems. The Intelligence Collection 
Process (ICP) is broken down into strictly defined subprocesses. Each subsystem within a 
CCOP system will perform one or more of these ICP subprocesses. For example, a CCOP 
system might have a subsystem that was responsible for carrying out subprocesses P3 and 
P4 from the table below. Such responsibilities would include all actions listed for P3 and 


P4 on the right had side of Table 1. 
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Subprocess Name Descriptio: 
il Review Request b Determine if collection capability is 
available 
* Determine if further direction or info 
required 
ee * Review directives and target information to 
: determine type/category of target 
ahaa lial 
CCcOP resources to each target 
iS a 
* Observe sensor data for target cues 


fe) Target Data Acquisition/Capture | * Audio Routing 
: * Record/Capture Data 


Target Data Processing * Demodulate, decrypt, direction find (DF), or 
Geo-locate 
* Translate 
i isl 
data 
«ala laeeeereieenasramna 
e Generation 
| aii * Check format, accuracy and adherence to 
tasking, regulations and laws 
P10 | Transmit Report * Transmit via secure voice radio, secure 
internet relay chat, US Message Traffic 
Format 


Table 1. The Intelligence Collection Process (ICP) 





The actions associated with the subprocesses can then be broken down with even 
more granularity into the individual components that are required to make that particular 
subprocess function. As an example, the components involved within subprocess P6 in 


Table 1, “Target Data Processing” is as follows. 


[pe [Target DataProcessing SSS 

|_| Human-based (no automationrequired) 

| | Manual copydirectlyintoreport 
Demodulate 

| Decrypt  — —s—“‘“‘CSCS*C*C™C™C™C*SC 


Special processing 





= 
= 
7 
= 
Ld 
= 
= 
| | AlIT-based 
ii 
= 
7 
= 
_ 
i 
= 


Table 2. Process P6 Actions 
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There are also humans involved in the ICP and since the output from all of the 
CCOP systems is the same, the members of the crew assigned to operate the CCOP 
system are considered in the KVA method as well. As an example: as crew members are 
assigned to their respective ICP processes, not only might several crew members be 
involved in the same process but each crew member is also involved in multiple 
processes as well. This complexity can be seen when analyzing the performance of 
CCOP systems under different crews. A more efficient and knowledgeable crew often 
knows how to make the best use of the system they are using and so the return for that 
system is higher. This is also a good example of why detailed analysis is required when 
reviewing KVA analysis results. Factors such as crew experience are difficult to capture 


in an algorithm but can be explained through analysis. An example of how humans 


integrate into a CCOP system can be seen in Table 3. 


Processes 


On 
gOp 


Div LPO 
| SigOpt4 | 
| ComOpt Tt 
| ComOp2 | lt 
| ComOps Tt 





Table 3. | Sample Crew from Ship A 


Table 4 shows the CCOP system breakdown for Ship A. There are six different 
CCOP systems, which, for the sake of classification issues, are represented below by 
letters A-F. These systems work together to accomplish the ICP processes and 


subprocesses, further complicating the inner workings of the KVA method. 
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[pi —s—~—t*~=“‘é*d ReiewRequest/Tasking = TAC 
lp2 tstsisd Determine Op/Equip Mix lA 
[ps ——SSSC=d Input Search Function/Coverage Plan [A __| 
|pa_——~=*‘“‘*C*d Se-aarcch/Coollection Process TAC 
ips tsts—sidC Target Data Acquisition/Capture iA 
| Pst] SCSignalTypet—— (ss —CiTC 
| PS? | ~SCSignalType2_——— —“(‘isdTC CC 
| ps3 i| SCS CSigalType3—— —“‘“‘L]S OC 
| ese C| ~SCSignalType4 = CC] CEC 
[Pe ——=*=“‘;*C*di Care Data Processing == | CC 
| Pet | ~SCSSignalTypet1—— tC‘ 
| 2 C| SC‘ Type2)—— is—“‘ SWCrTC OCC 
| pes S| CS CSignalType3—— t—“‘L]SO COCO 
| Pee | ~SCSignalType4 =—— “aes —“(isdTC CEC 
[p7_———~—~*=“‘;*C*darget@DataAnalysis | 
| Prt id] SS CSignalTypet — i Cd CC 
| 72 i| SCSignalType2_— C(t CC 
| ers | SS CSignalType3—— “(ists CCC 
| pra Sid| SCS CSignalType4 tC tC EEC 
|ps_——S—~SC~C~C*d Format Data for Report Generation = |AF 
[po ——=—t*‘*d QC Report ee —“(i‘“‘;C*SC*d AA 
[pio ——~—“*;‘sSC*diCrramsmitReport == Ci—i‘ir CC 





Table 4. | Ship A CCOP System Breakdown 


Just as processes and subprocess can be broken down into the actions that 
comprise them, so too can systems also be broken down into their related components. 
Below is CCOP system A from Table 4 as an example to show the complexity of said 


system and the interdependence of these systems upon each other. 


[CCOPA(Example) OO OCCOCOCOCOC™~CSOO..CCC“‘(;A€Né;NSTCC‘#d’ 
Radio Frequency Management System 
Signal Acquisition System 
Audio Distribution System e Audio Routing & Recording 


intermediate Frequency Signal Processing * Spectrum Display Operations 


System ¢ Signal Processing Applications 
Common Cryptologic Workstation (CCWS) * Database Operations 
JMCIS Applications 
Cryptologic Unified Build Applications 
Microsoft Applications 
Signal Processing Applications 





Table 5. | Ship A CCOP System A Components 
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2. Data Required for Analysis 


The thesis work done by LCDR Rios, LT Clapp and LT Lambeth has proven the 
concept of generating ROI estimates based on KVA analysis. The next step in moving 
KVA analysis research forward is finding a data collection method that facilitates the 
inject of real world data from US Naval vessels afloat into the KVA method. The 
common form of output used in determining the value of a CCOP system is the 
generation of KL reports. It would therefore be the simplest solution if a means could be 
devised by which KL reports were fed directly into the KVA analysis engine as they were 
issued. However, there is a fundamental problem with that approach, namely that the KL 


does not necessarily contain all of the data needed to accomplish accurate KVA analysis. 


Since the KL is a highly classified report, the specifics of what it contains cannot 
be discussed here. As it turns out, the specifics of what is contained in the actual KL 
report is essentially irrelevant to the KVA process anyway. The only thing the KVA 
analysis cares about is the fact that a KL was sent, not the actual content of the message. 
The reason for this is that as far as the KVA analysis is concerned the KL acts more like a 
counter than a data delivery device. The fact that a KL was sent is more important than 
the content of the message. A KL being sent indicates that a CCOP process fired and 
value was gained from the system. The end result is that there is additional information 
that will be required if KVA analysis is to be conducted using KL reports as its 


motivation. 


There are two critical and two non-critical data types that either do not appear in a 
KL report or are optional fields in the output and thus cannot be counted on to be present. 
The first two are CCOP systems used and total work time. The others are the latitude and 


longitude of the system at time of collect and the KL date time group. 
a. KL Date Time Group 


This data type is used for ease of correlating the KVA data with the actual 
KL report. It is not required for KVA analysis but it is necessary if any correlation is to 


be done in the future between the ROI data and the operational data. 


TZ. 


b. CCOP Systems Used 


Which specific CCOP systems are used in the generation of a KL report is 
a critical piece of information in the KVA analysis process. Without it, an accurate cost 
estimate cannot be obtained. The integration of CCOP systems is shown in Table 4 and 
the movement of information through the collection and reporting process often requires 
the services of numerous systems in order to transition from intelligence collection to a 


KL report. It is important to capture each of the systems involved. 


c. Total Work Time 


Another critical piece of information required in the accurate calculation is 
the time that each CCOP subsystem is used to produce the KL output. Essentially, the 
amount of time a particular resource within each subsystem is occupied such that the 


particular resource cannot be used to service another system. 


There is a potentially significant difference between total work time and 
total elapsed time. The total time the intelligence is in the CCOP system is not necessarily 
what is desired for KVA analysis. What is needed to conduct an accurate KVA analysis is 
the amount of time during which CCOP sub-system work is actually being done. A 
simplified example of this difference would occur when 10 minutes of analytic work is 
done, at which point the operator leaves to go to the bathroom for 10 minutes and then 
returns to finish the final 5 minutes of work. The total elapsed time would be 25 minutes 
while the total work time would be 15 minutes. Again, it is the total work time that is of 


consequence to KVA analysis. 


d. Latitude and Longitude 


The latitude and longitude (lat/long) of the system at the time of the 
collect is not a required piece of information in order to conduct KVA analysis. However, 
it does have practical and potentially important secondary benefits. The importance of 
this data type is not for the actual KVA analysis but rather for the secondary analysis that 
can be conducted on the ROI data. 
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The most useful immediate analytic capability that lat/long data provides 
is Maximized in a near real-time (NRT) data scenario. If KVA analysis can be done in 
NRT utilizing lat/long data then it allows for mapping capabilities that would be useful to 
the operational planner on the fly. One possible implementation of such a mapping 
capability would reside with the Cryptologic Resource Coordinator (CRC) and his ability 
to maintain situational awareness of the CCOP systems under his authority. One way this 
could be done would be through a central repository where ROI data is stored and 
translated into a visual display. This display could show the CRC at a glance the 
effectiveness of all of the CCOP systems under his authority. The display would assist 
the operational decision maker (the CRC) in assessing which systems are most effective 
against certain targets, where the best locations are for reception, which systems are 
performing at, below, or above expectations and where potential problems might be. 
Once this concept is expanded beyond the limitations of the KL report and beyond the 
scope of strictly CCOP systems, the CRC would have at his disposal a complete picture 
of the location and health of all of his assets. He could potentially also recognize 


equipment or training deficiencies based on fluctuations in expected ROI for a system. 


Another benefit for recording lat/long data is in the generation of historical 
analysis reports such as a post deployment ROI report. Using a Google Earth™ type of 
interface a summary report could potentially show a map of the entire deployment from 
which you could analyze the performance of specific systems based on geographical 
location, range from shore or any other number of factors. It would also help with 
analysis of circumstantial oddities such as long transits, where systems are often idle due 


to no fault of their own. 
e. Data Capture Form 


Since the KL report does not reliably provide the data types required for 
KVA analysis, a second means of data capture is required to capture them. A simple form 
should be all that is needed, as the number of elements of required data is relatively 


small. The form below is offered as a solution for capturing the required data. 
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KL Date Time Group 
231905Z JUN 09 


| ow | or | Eb | Eh 


Total work time 
Start Finish 
231900Z JUN 09 231905Z JUN 09 

















Table 6. | Data Capture Form 


3. Assumptions 


In order to facilitate crewmember data collection it is important to find ways to 
minimize the impact on crew activities while at the same time maintaining the integrity of 
the data. This results in the generation of a small list of logical assumptions that will ease 
the workload on both ship’s company and the KVA data analyst. Listed first are 
assumptions carried forward from the thesis research conducted by LCDR Rios, LT 
Clapp and LT Lambeth concerning the underlying assumptions governing the KVA 
process. The KVA analysis assumptions are included for the benefit of the reader and can 
be studied in depth in the thesis work conducted by LCDR Rios. Following that are the 


assumptions made during this current research. 
a. KVA Assumptions 


The assumptions that provide the foundation for KVA analysis have not 


changed and are provided here as background for the reader. 


e Humans and technology in organizations take inputs and change 


them into outputs through core processes. 
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e By describing all process outputs in common units (i.e., the 
knowledge required to produce the outputs) it is possible to assign 
revenue, as well as cost, to those processes at any given point in 


time. 


e All outputs can be described in terms of the time required to learn 


how to produce them. 


e Learning Time is measured in common units of time and is also a 
surrogate for knowledge. Thus, units of Learning Time can also be 


called Common Units of Output (K). 


e Having a common unit of output makes it possible to compare all 
outputs in terms of cost per unit as well as price per unit, since 


revenue can now be assigned at the sub-organizational level. 


e Once cost and revenue stream have been assigned to sub- 
organizational outputs, normal accounting and financial 


performance and profitability metrics can be applied to them. 
b. Embedded Knowledge Estimates 


The estimates used to measure the amount of knowledge embedded in a 
CCOP system has not appreciably changed since the thesis work conducted by LT Clapp 
and LT Lambeth. This section is provided to give the reader an understanding of how the 


amount of knowledge embedded in a particular sub-process is determined. 


According to LCDR Rios, the knowledge “embedded in information 
technology (IT) systems can be derived by averaging the time it would take an average 
learner to learn how to produce the outputs produced by the IT systems in a single 
subprocess output cycle. CCOP systems are highly complex and at times, comprise 
multiple components with varying functions. To estimate the time to learn of a single 
CCOP system, the components were analyzed individually. Academic authorities on the 
functions performed by each were consulted to determine the length of time it would take 


an average learner (assuming at least a Bachelor’s of Science (B.S.) degree in Electrical 
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Engineering or in a field related to the component) to learn how to produce the IT 
outputs. In this case, subject matter experts in the functional fields of each system were 
consulted to estimate the IT time to learn.”!9 An example of this process is included as 


Figure 1. 





Figure 1. Example of Embedded Knowledge in a CCOP System 


Cc. Total Work Time Estimates 


Ideally, there would be a start time and a stop time for each individual 
system involved in the generation of the KL report. Information about system use times 
would give an exact representation of how long resources were unavailable due to being 
consumed with the task in question. However, such granularity would require the 


operator using the equipment to record all such data for every system involved from start 





19 Cesar G. Rios, Jr., "Return on Investment Analysis of Information Warfare Systems" (MS thesis, 
Naval Postgraduate School, 2005). 
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to finish. As was shown previously, this process could involve numerous different 
systems, some of which trigger each other without even signaling to the operator that a 


handoff has occurred. 


The difficulty inherent in trying to time all of these events makes the job 
of recording all such information tedious, time consuming and even beyond the 
operator’s abilities in some instances. The best way to gather the desired information 
without placing undue burden on the ship’s crew is to collect only the start and stop times 
for the entire process from collection of intelligence to KL report release. The level of 
detail lost by taking this approach can be minimized by using historical data to generate 
averages for the times required by each system. These averages should be reassessed at 
specified intervals to ensure that they remain accurate. The loss in granularity through 
this process is compensated for by the reduction in human error and the amount of data 
that will be received. There is also an intangible benefit to simplifying this process, one 
that cannot be stressed enough, and that is the willingness for the operator to take the 
time to gather the data needed for KVA analysis. Any way that can be devised to simplify 
and shorten the process for the operator reduces the cost exacted on their time and 


increases the likelihood of cooperation and compliance. 


d. Start and Stop Time Calculation 


Due to the nature of intelligence collection there are many aspects of the 
collection and reporting process that can be affected by the interpretation of an individual 
crewmember. One such interpretative bias is when the clock starts and when it stops. In 
other words at what point can it be said that the product has officially entered the system 
and when it leaves. The start and stop data collection techniques provide information on 


how long resources are unavailable to service other CCOP processes. 


The start time has two possible definitions. The first definition is from the 
moment of collection and the second is from the moment of recognition. The difference 
is that the former always happens first while the latter may correspond or it may happen 
later. In U.S. Naval intelligence gathering the clock always starts upon recognition by the 


system or by the operator that the collection is of value and should be reported. 
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Therefore, it makes sense to follow suit with the start time for KVA analysis. It is true 
that system resources are dedicated starting at moment of first collection, however that 
time can be extremely difficult to determine and is open to interpretation by the operator. 
The moment of recognition is concrete and easily recordable, resulting in a much greater 


degree of accuracy. 


Like start time, stop time also has two possible definitions. The first is the 
moment of message release and the second is the time at which the KL report is received 
by the consumer. For many of the same reasons it is much more accurate to record 
message release time as it is directly controlled by the operator and is easily determined. 
Message reception by the consumer is affected by many factors outside the scope of the 
system being considered. Some potential factors of influence are message precedence, 
relay station outages and amount of message traffic on the lines at the time. For all of 


these reasons the moment of message release is the best choice for stop time. 


To summarize, this study assumes a start time of the moment of 
recognition of a piece of intelligence and a stop time of the moment of KL report released 


by the ship’s crew. 
e. Total Elapsed Time vs. Total Work Time 


There can be a significant difference between the total amount of time that 
elapses and the total amount of time a system spends working. If an operator takes a 
break between the moment of recognition, which has been determined as the start time, 
and the moment of message release, which has been determined as the end time, then 
there can be significant error in the resulting ROI calculation. However, given personal 
experiences of those who have worked as members of ship’s company and the time 
constraints placed on KL release by the classified documents governing KL reports, it is 
very unusual for such a delay to take place. In fact, it is so unlikely that this research 
assumes that as far as KL reports are concerned, total elapsed time and total work time 


are equal. 
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D. THREE OPTIONS FOR CONDUCTING KVA ANALYSIS 


The thesis research conducted by LCDR Rios, LT Clapp and LT Lambeth 
established a rigorous method for generating ROI estimates for CCOP systems using 
KVA analysis. What has been missing is a functional way to collect, retrieve and analyze 
data from US Navy units afloat. This research will develop and analyze three possible 


solutions to this problem. 
i Option 1—Standalone Laptop 


In this method, the researcher will provide a laptop to the Ship’s Signal 
Exploitation Space (SSES) Division Officer (DIVO) with a database installed and 
preconfigured with the Data Capture Form discussed previously. This laptop will 
function as a standalone computer dedicated to KVA data entry. In this method the 
operator will be required to enter the specified KVA data into the standalone after the KL 
report has been released. The effort required of the member of ship’s company for data 
entry is minimal and is comprised of the four extra pieces of information outlined in the 


Data Capture Form. 


At the completion of the reporting period the data that is resident on the laptop 
will be retrieved from the ship to facilitate analysis of the captured data. There exist two 
primary means by which this can be accomplished. The first and easiest to accomplish is 
for the laptop to be collected from the ship at the conclusion of the deployment. The 
second and more difficult is to periodically download the data from the laptop onto a 
disk, move it over to the ship’s network and send it back via File Transfer Protocol (FTP) 
to either NPS or SPAWAR. This could be done at any set interval (i.e., daily which is 
most desirable or weekly which is most likely) but would require additional effort from 
SSES personnel. This method is preferable and if an agreement could be reached for 


daily transmission would result in a close to NRT effect. 


If physical collection of the laptop is employed then the laptop will be collected at 
the conclusion of deployment. At that point the data can be retrieved and analyzed using 


KVA analysis to provide historical ROI data for the entire deployment. If the FTP 


ZS 


collection method is employed then an analyst at either NPS or SPAWAR can receive the 
data at the agreed upon time intervals, run the data through the KVA analysis process and 


provide timely feedback to the ship on CCOP system performance. 


a. Advantages 


The main advantage of this option is the ease of setup and the low cost. 
This method could be employed almost immediately on one ship for no more than the 
cost of a laptop and a plane ticket for someone to fly to the ship to explain what is 
required of them. The laptop could be picked up or shipped at the conclusion of 
operations and analysis can be conducted at either NPS or SPAWAR. This data is 
perfectly suited for historical analysis and can be used to show how each CCOP system 


performed during the duration of the deployment. 


b. Disadvantages 


Since this option uses a standalone laptop it is going to require the 
operator in SSES to physically get up from his workstation after sending the KL and 
enter the data into a separate terminal creating yet one more task on top of an already 
potentially busy schedule. The problem with this is that such a situation could lend itself 
to the operator putting off and forgetting to enter the additional data or just not having the 
time to do so. Also, if the data is not collected at least daily, then the NRT aspect of the 


analysis is lost along with all the advantages such a capability provides. 


2. Option 2—Additional Message 


A second option that exists as a possibility is to create a new message that can be 
sent through the Navy’s message handling system via the same means as the KL itself. In 
such a situation, the operator would complete his KL report just as he normally would. 
After he finished he would pull up a message mask on the same computer he had just 
used and cut and paste any relevant information needed from the KL report. He would 


then fill in the remaining required information as outlined in the Data Capture Form and 
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release the message in much the same way as he had just released the KL. At this point, 


the operator has completed his role in the process and need not think of it again. 


There are two possible methods of retrieving and analyzing the data. One is a 
manual method, which is more time consuming and human intensive, the other of which 


is essentially completely automated. 


In the manual method, an analyst would manually pull the message traffic off of 
the message server each day, run the analysis, generate any reports or graphs that are 
needed and then send that output to whoever desires it. In the second method, the data 
would be pulled out of the message traffic stream in exactly the same fashion as the KL 
and stored in a database. A KVA analysis server can pull the data and run its analysis. 
The server can then generate any reports of any type and content required and email those 


out to a preset distribution list. 
a. Advantages 


The first advantage of Option 2 — Additional Message is that it requires 
very little of the operator’s time, thus increasing the likelihood that procedures will be 
followed as expected and the data set will be complete. The less extra effort that is 
required of SSES personnel, the more likely they are to fully comply. Option 2- 
Additional Message also eliminates the necessity for the operator to send the data via 


FTP or any other method where yet another extra step is added. 


Data retrieval is possible whenever the receiving node makes a request, 
taking the responsibility away from the ship. If manual retrieval is being conducted then 
it could certainly be done once or even twice a day. As soon as the KVA analysts are 
trained in how to enter the KVA data and generate the desired output it could be put in 
place, creating a relatively simple training requirement for the analyst doing the work. If 
automated retrieval is done then it can be done in NRT. An additional benefit with the 
automated method is a reduction in human error. Once the scripts and programs are in 
place to generate an automated response it will require minimal human interference, 


reducing associated manpower costs as well. 
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b. Disadvantages 


Disadvantages of Option 2—Additional Message reside almost entirely on 
the data retrieval and analysis side. If manual analysis is done because of the lower initial 
costs then problems caused by human involvement must be dealt with such as increased 
error rates, likely loss of NRT capability and manpower costs. However, if an automated 
solution is selected then a different set of problems arise involving higher upfront costs. 
A dedicated server will be required to pull the required messages from message traffic. A 
text parser will then need to be developed in order to translate the message into the 
correct input format for the KVA analysis. After the analysis is done and the appropriate 
reports and graphs have been generated the server will need to disseminate those to the 
appropriate individuals via email. Finally, a script will need to be developed that executes 
all of the aforementioned functions. However, none of these problems are too difficult 


and once overcome will require only routine maintenance and periodic tweaks. 


3. Option 3—Changing the KL Report Format 


The third option is to make a change to the format of the actual KL itself. Since it 
already contains the KL date time group the change would consist of making the three 
remaining pieces of information (systems used, the start/stop times and the lat/long) 
required data fields on the KL mask. This option functions very much like Option 2- 
Additional Message, with the exception that the operator would not have to do any extra 
work whatsoever. Since the required information would be a part of the KL report the 


operator need not be involved further. 


Data retrieval would function in essentially the same manner as in Option 2- 
Additional Message, with the exception that the message being pulled would be the KL 


itself and not an alternate message. 


a. Advantages 


Aside from the advantages for Option 2—Additional Message, there are 


other significant benefits. A successful change to the format of the KL would mean that 
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everyone who writes a KL would have to use the new format. This would mean that, as 
long as a platform issued KL reports, data would become available for every system, not 
just CCOP, used on all of those units around the world. There would be no need to 
request data from certain ships as all ships would be feeding the data automatically. It 
would be a relatively easy task to expand the scope of KVA analysis from CCOP systems 


to other systems as well. 
b. Disadvantages 


While the potential benefits are significant, the disadvantages are also 
significant. The format for the KL report is governed by national-level policy and would 
be extremely difficult to change. Convincing national level policy makers to alter 
doctrine on behalf of an NPS thesis would be a daunting challenge that could take years 


with little hope of success. 


Another problem arises from the likelihood that the full ramifications of 
changing the format of the KL report cannot be fully known until the change takes place. 
KL reports are fed into national level databases automatically. Changing the format for 
the root message could force all other customers of KL reports to also have to change 
their systems, resulting in potentially high reprogramming costs. It would, at the very 
least, require changes to the training that operators receive and all costs associated with 


reeducating the fleet. 
E. EXAMPLE OF USAGE: POST-DEPLOYMENT REPORT 


Most project approvals in the Navy’s marketplace, involves answering the 
question “What does it do for me?” In anticipation of just such a question, the following 
is provided for the reader as a simple example of what might be expected as a potential 


value added product resulting from ROI data generated by KVA analysis. 


Such a report should include historical visualization of how each system 
performed per deployment, a map showing where CCOP system activity took place 


throughout the deployment and a detailed performance analysis to ensure fair 
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treatment of the given CCOPs, accounting for transit time, range requirements, system 


downtime, etc. Possible content for such a report follows. 


i Deployment Dates by Fleet 


This section would provide pertinent data about how long the unit spent in a 
specific Area Of Responsibility (AOR). For example, to indicate that part of the 
deployment was spent in the Sixth Fleet AOR and the rest in the Fifth Fleet AOR the 


report might say something like this. 


e C6F-45 days, C5 —188 days, Total—233 days 


2. Operations Summary 


Here is where data concerning any major operations and exercises would go. 
Anything that might lend understanding to where and how the assets of concern were 
used is important to mention. Examples would be Operation Enduring Freedom, 


Operation Iraqi Freedom, Maritime Security, etc. 


3s Cryptologic Capabilities 


This list should be as extensive as possible as it will help explain things in the 
detailed analysis that will come later. It should include such things as units involved, 


systems available, ranges, personnel numbers, training levels, etc. 


4. Collection Priorities 


Similar to capabilities, this list should be as comprehensive as possible and should 


include such things as countries, platforms, systems, etc. 


5. Reporting Statistics (KLs) 


This is where deployment statistics for the systems in question would go. There 
are near infinite ways to break this section down but some possibilities are total number, 


number for each CCOP system, number by geographic region, etc. 
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6. Map Display 


The visual demonstration of the deployment will likely dominate the report and as 
such should provide as much illumination into the performance of the systems under 
question as possible. One possible way to use this feature is to place pins along the 


deployment route showing the location of each collect. 
7: Analysis 


This is the most important section of the report since it is the opportunity to 
support or rebut the systems in question. It is here that the report would take into 
consideration any factors not made apparent by statistics or the map. Generally, the 
analysis will address things that might lower a CCOP system’s ROI estimate unfairly 
such as periods of long transit where collection is impossible, equipment failures, range 
limitations, interference, etc. Carrying out a detailed analysis of the ROI data will provide 


a perspective on the raw numbers that decision makers need to make informed decisions. 
F. CHOOSING THE RIGHT OPTION 


Each of the three options (Option 1—Standalone Laptop, Option 2—Additional 
Message and Option 3—Changing the KL Report Format) can collect, retrieve and analyze 
the data. For the purposes of this thesis however, Option 3—Changing the KL Report 
Format is being removed from consideration. The process for achieving organizational 


adoption is time consuming and resource intensive offsetting potential economic benefits. 


Option 1—Standalone Laptop and Option 2—Additional Message function in 
fundamentally different ways. Option | has the benefit of being easily deployable on a 
very limited basis very quickly. It is ideal for a trial run on a single ship. It can be 
deployed on a ship with very little effort and then collected at the end of the deployment 
and taken home for analysis. The cost is one laptop, approximately 2 hours of training for 
the operators on the ship and approximately the same amount of training for the KVA 
analyst after the deployment is finished. What is lost is the ability to receive data in NRT 
and all of the benefits that go along with NRT data analysis 
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On the other hand, with Option 2—Additional Message NRT data analysis and the 
situational awareness inherent in NRT data flow are attained. Also gained is a system that 
is easier to scale up. Once operators on the ship and the analysts back stateside have been 
trained, another ship can be added to the data stream simply by training the second ship’s 
operators. The down side to having a NRT capability is that the process needs to be 


automated and all of the upfront costs associated with that process have to be accepted. 


With all of the discussed factors under consideration, the best course of action is 
to use Option 1—Standalone Laptop in the immediate. It makes a great first run test bed 
with little cost risk should anything go wrong. While Option 1—Standalone Laptop is 
running its course, further implementation research should continue on Option 2- 
Additional Research so that it is ready to be implemented at the successful conclusion of 
Option 1—Standalone Laptop. This course of action would allow for real world data 


collection to begin while preparation for a long-term solution continues. 
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Iii. CONCLUSIONS 


America is currently dealing with an economy that is in recession. It is more 
important than ever to maximize the benefits received from the expenditure of taxpayer 
dollars to defend our nation. As government agents, it is important that members of the 
Department of Defense and specifically the Navy make the best use of the resources we 
are afforded. Navy ISR is a critical piece of our ability to both attack and defend and as 
such deserves the absolute best we can provide. ROI analysis is an attempt to help 
decision makers equip themselves with the best information they can possibly get in 


order to make informed decisions concerning the stewardship of our resources. 


This research project represents the extension of an existing KVA method that has 
operated within a static environment into a dynamic platform that can function among 
emerging DoD needs. This thesis presents a set of options and proposes a capability to 
gather data from US Naval vessels afloat. That collected data will then be used to conduct 
KVA analysis in order to generate ROI estimates. The ROI estimates will provide key 
decision makers with a valuable and proven method to evaluate technology options in the 


acquisition process. 


The combination of options that have been recommended in this research will 
allow for valuable data collection to begin now, while preparing for a larger scale and 
more permanent solution in the near future. This process will begin to provide key 
decision makers with the valuable tools they need to make budgetary decisions. The 
output produced by this process will help shape the future of Navy systems acquisitions 


for years to come. 


The requirement for Navy ISR is going to increase as time goes by and warfare 
becomes more and more unconventional. Additionally, budgetary constraints will always 
be present and resources will always be limited. Given those two factors, it is imperative 


to the continued growth and development of ISR technology that money be spent wisely 
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and with clarity. Applying the tools provided by this research will help track the value of 
current technologies and provide decision makers with the ability to make better 


informed investment decisions. 
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IV. RECOMMENDATIONS 


Measuring systems effectiveness based the output of a KL report does not provide 
us with a comprehensive understanding of CCOP system performance or the ability to 
make informed procurement decisions. The limitation is supported by the fact that the 
issuance of KL reports can vary greatly depending on the personalities of the operators 
involved and thus does not provide an objective measure of CCOP system performance. 
In July of 2009 a group of CCOP system managers met at SPAWAR in San Diego, CA to 
discuss a detailed data set that would provide a larger base of historical data and a more 
reliable stream of new CCOP system performance data. This data set came from an 
automated reporting system that is tied directly into the CCOP system, reducing and 
possibly even eliminating the need for human intervention within the data feedback loop. 
This would eliminate any inconvenience imposed on the ship’s crew, reduce the potential 
for operator data errors, and also eliminate the need for any additional KVA data 
collection equipment installation or implementation procedures. Additionally, once the 
procedures for analyzing automated data feeds have been established it should be 


relatively straightforward to attain a near real time data stream. 


According to the information distributed at the San Diego meeting on collecting 
KVA data, it appears that all required data needed to conduct KVA analysis were present 
in the CCOP automated data feed. Further research needs to be done to verify that all 
required KVA data input is present in order to conduct a reliable and valid KVA analysis. 
Once the presence of all those elements describes in the Data Capture Form has been 
verified, both the data collection process and the KVA analysis process need to be 
reassessed to ensure that all of the previously established assumptions and procedures are 
valid. Specifically, there needs to be research done that answers the question “What is the 
relative value of various automated reports compared to the value of human-in-the-loop 
reports such as the KL.” There needs to be research performed on defining the 
fundamental differences between human generated and automated reporting with a focus 


on developing a means of objectively comparing the differences and similarities between 
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them. It is possible that the two different types of reports may be used to validate and/or 
calibrate each other, especially if there is any overlap or duplication of effort in the 


generation of each report. 


Additionally, this research is now mature enough to facilitate looking for other 
venues to apply KVA analysis to OPNAV information technology performance 
assessment. An ROI estimate based on KVA analysis is a valuable acquisition decision 
making tool and can be used to aid decision makers in making informed acquisition and 
portfolio optimization decisions. The next phase of research into applications of KVA 


analysis needs to take these possible extensions into account. 


The next step in implementing KVA analysis to generate ROI estimates is to 
implement Option 1—‘Standalone Laptop” as described previously. The reasons for 
selecting this option are as follows: low set up costs and ease of implementation for a trial 
run. Once a ship is identified and committed to the trial implementation, prior to vessel 
deployment a laptop computer with all required software installed needs to be delivered 
to the participating vessel and operators need to be trained on which data need to be input 
into the software interface. When the vessel returns from deployment the laptop can 
either be picked up or shipped back to either NPS or SPAWAR. The next steps also 
involve working with the SPAWAR team to set up data collection procedures at the Point 


Loma facilities using the Gauss Soft KVA performance accounting software. 


There are many potential benefits that can be attained from a trial implementation. 
Even though continuing research on automated reports as well as in automating the 
collection, retrieval and analysis process, the data collected on the laptop will enable the 
application of the KVA data collection and analysis using performance accounting 
software. While “Option 1” is not an optimal long-term solution, the software 
implementation learning curve benefits make it a worthwhile effort. During the time 
period when the data generated by implementing “Option 1” is being collected it is 
recommended that research be conducted on how to accommodate the type and volume 
of automated data. Additionally, work should continue on automating the processes 


associated with retrieving, analyzing and reporting readily available KVA data. 
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To summarize, the results of this study suggest that “Option-1” standalone laptop 
be implemented to enable fine-tuning the process of collecting, retrieving and analyzing 
real world data from US Naval vessels afloat to facilitate KVA analysis. In addition, 
continuing research should focus on automating data collection procedures and 


expanding use of KL report data set. 
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APPENDIX A. GAUSSSOFT OVERVIEW 


GaussSoft is the analysis tool by which the KVA analysis is done. This GAUSS 


overview provided courtesy of GaussSoft, Inc. <http:www.gausssoft.com> 


GAUSS is a line of software created by GaussSoft, Inc., a privately held U.S. 
corporation founded in 1993, with headquarters in San Jose, California and an extended 


presence with offices and partners in North America, Europe and Latin America. 


GaussSoft delivers scalable Business Intelligence solutions of unrivaled 
performance, enabling large and medium-sized companies to control and reduce the cost 
of enterprise operations, increase profitability and improve organizational productivity by 


providing unsurpassed flexibility, scalability and ease of use. 


GaussSoft’s solutions are built on an integrated suite of high performance 
products for Profit and Cost Analysis, Multidimensional Query, and Activity Reporting 


that are scalable, function-rich, and easy to use. 


GaussSoft has installed performance intelligence solutions in over 200 enterprise 
and consulting companies all around the world, including telecommunication, banking, 
manufacturing and agribusiness firms and government organizations. They have been 
implemented in customer premises by leading consulting firms including Deloitte, 


KPMG and Price. 
GaussSoft suite includes: 


Gauss—-Profit and Cost Allocation Engine: This strategic decision-making and 
analysis solution enables companies to know which products, services, and customers are 
making profits and which are not. Using different value and costing methodologies this 
solution helps reduce and control the cost of enterprise operations, increase profitability 


and improve organizational productivity. 


Gauss—KVA: Knowledge Value Added (KVA) is a methodology that allows any 
organization to calculate the economic performance of core processes by providing an 
objective way to allocate revenue to the processes at any level within the organization. 
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Knowing how much revenue corporate knowledge is producing, allows organizations to 


dramatically improve their effectiveness and efficiency. 


Gauss-—Planning: This enterprise collaborative solution allows thousands of users 
to perform corporate enterprise planning, including financial planning, budgeting and 
forecasting up to 10 times faster. When used with Gauss Profit and Gauss KVA, an 


organization can create plans optimized for profitability and value. 


Gauss-Radial Viewer: This is a Business Intelligence (BI) front-end with 
graphical interaction. This tool enables all End Users to create their own queries and 


professional looking reports from scratch —in seconds. 
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Figure 2. GaussSoft Accumulator View for USS GONZALES Case Study 
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Figure 3. GaussSoft Radial Viewer Report Design Screen 
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APPENDIX B. 


USS READINESS KVA ANALYSIS 





























CREW 1 PERSONNEL TIME SPENT PER PROCESS 
aS - 

Time in cat On-Job 

Service | Training | Training 
Operator | (Days) | (Days) | (Days) | Totals | Processes P41 P2 P3 P4 P5 PG PT Ps Pg P10 
Div Officer 730.00 16 292] 4,037 25.00% [35.00%] 
Div LPO 4124.50| 15 524 | 4,664 [2-7,9 10.00% 20.00%] 10.00%] 25.00%] = || ~—Ss—«5.00%| 
SigOp 1 1131.50] 30 486 30.00%| 20.00%] —10.00%| 10.00%] =| ~—— 10.00% 
SigOp 2 1131.50] 30 366 i 25.00%| 10.00% 
SigOp 3 1131.50] 30 325 25.00%| 15.00% 
ComOpt 4124.50] 20 325 10.00% 
ComOp2 1898.00] 20 219 10.00% 
ComOp3 1131.50] 20 184 10.00% 

































































































































































CCOP A Aggregated Time to Learn = Assumptions: 
CCOP B Time to Learn = (CCOP System Time to Learn is divided evenly over subprocesses in which they operate) 
CCOP C Time to Learn = 
CCOP D Time to Learn = 
CCOP E Time to Learn = 
CCOP F Time to Learn = 
Process | Other Tottur | Tott. for1 
Training | Relevant times% | Process 
ccoP ty tly TOTAL CCOP tur] Avg% |Automat'n| Output 
Sub-Process Name Assigned | (days) days) _| Tih (days) (days) |Automat'n’ days) (days) 
PI Review Request/Tasking A 20 [332 352 492| 25.00% 843.70 
P2 Determine Op/Equip Mix A 580 590 492| 10.00%] 550.91 | __ 4,082.34 
P3 [Input Search Function/Coverage Plan A 35 (637 672 492] 20.00%] 626.19] 1,163.54 
P4 Search/Collection Process A 35 (2347 2382 492] 35.00% 2,874.02 
P5 Target Data Acquisition/Capture A 16-1613 1629 492| 35.00% 1,664.42 
PSA Signal Type 1 B 312| 35.00% 426.00 
P5.2 Signal Type 2 Cc 198] 35.00%] 312.00 | 312.00 
P53 Signal Type 3 D 608] 35.00% 122.33 
P54 Signal Type 4 E 284| 35.00% 397.67 
P6 Target Data Processing 340 [805 1145 573 50.00%]  —s_—|| 
P61 Signal Type 1 B 312| 50.00% 455.18 
P62 Signal Type 2 c 198] 50.00% 341.18 
P6.3 Signal Type 3 D 608] 50.00% 751.52 751.52 
P64 Signal Type 4 E 284] 50.00% 426.85 
P7 Target Data Analysis 50 1367 1417 708 50.00% 
PTA Signal Type 1 B 312| 50.00% 489.09 
P72 Signal Type 2 c 198] 50.00% 375.09 
P7.3 Signal Type 3 D 608) 50.00% 785.42 
PTA Signal Type 4 E 284] 50.00% 460.76 
PB Format Data for Report Generation AF 10 [5718 5728 2864 682) 50.00% 6,410.10 
Pg QC Report AF 30-(609 639 575 682| 10.00% 1,320.56 
P10 Transmit Report F 14 (635 649 97 190] 85.00% 839.36 
560 8757 22,141.11 
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Avg cost for IT Fixed Infrastructure (annual) = 
|All other fixed costs (annual) = 
Length of Sample Pd as % of Year = 
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743,571.27 
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Please note that the floor for ROKA is -100% (e.g., zero return on knowledge assets) [Please note that the floor for ROKA is -100% (e.g., Zero return on knowledge assets) 
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KVA Metrics for CCOP EK KVA Metrics for CCOP F K 
b-Process | ROK as b-Process 
tame” | ator | roxas | nome | nom Mase 
Receive! Receive/ 
Review Review 
Request! Request! 
PI Tasking PI Tasking 
Determine ; Determine 
P2 Op/Equip Mix P2 Op/Equip Mix 
Load Search 
Func/ Load Search 
Coverage Func! Coverage 
P3 Plan P3 Plan 
Search! i | Search! 
PS Collection P4 Collection 
Target Data Target Data 
Acquisition/Ca Acquisition/Cap 
P5 pture 0.90 89.76% 11.41%] -10.24%| P5 ture 
Target Data lees Target Data 
P6 Processing 0.96 96.35% -3.79%| P6 Processing 
Target Data ze Target Data 
PT Analysis 1.04 104.00%| 3.85% PT Analysis 
Format Data Format Data for 
for Report Report 
PB Generation PB Generation 3.08 308.23% 
Pg QC Report |__| Pg QC Report 0.65 64.82% 
Transmit I 
P10 Report P10 Transmit Report 1.29 128.99%| z 
Metrics for Aggregated [2.90] 290.40%]-11.36%] _—-9.90%| Metrics forAggregated | _ 5.02 502.05%| 38.76% 

















$950,000 $8,000 $5,000 —— 
$175,000 | 2.000 -Ss| 85.000 


| $600,000 000 2 000 $5,000 a 


———— 000 $6,000 $7.500 PB 
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*System is not cross- 
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“Training not provided 
by CCOP 
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Coverage Plan Creation’ Management 
Control and Processing System 
Database Operations 

JMCIS Applications 

Microsoft Applications 

KL Writer 

Other CUB Applications 

basic RF 

EM theo: 

Basic Comms Theo! 
Propogation Theo! 

Antenna Theo! 

Basic Radio DF 

SCI Network Interface 
TDOA/FDOA 

Geolocation processing 

TCP/IP Communications 

VPN 

ALE 











































Near Real-time Signals Analysis 
RF Routing 

Ship navigation interface 
National Asset interface 

RF Management System 

Signal Acquisition System 
Audio Distribution & Recording 
Spectrum Display Operations 
Signal Processing Applications 
Demodulation/Decoding 
Audio/Visual Analysis 

Digital Signal Processing/Wireless Processing 
Mail Server/Exchange 

Data Encryption 
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CREW 2 PERSONNEL TIME SPENT PER PROCESS 
Pre- 
Time in ment | OnJob 
Service | Training | Training Assigned to 
rator Totals | Processes P41 P2 P3 P4 P5 P6 PT PS P9 P10 
Div Officer | _ 730.00) 46{ 292 | 4,037 [1,29 40.00% 25.00% [i | | 35.00% 
DivLPO | 4124.50) 16{ 524] 4,664 [2.7, 9 [10.00%]  10.00%| 20.00%] 20.00%] 10.00%] 26.00%] | 5.00%] 
SigOp 1 4124.50 30 486 | 4.644 |3-7.9 30.00%] 20.00%] 10.00%! 10.00%| =| 10.00% 
SigOp 2 1898.00) 30 366 | 2.294 [47 50.00%| 25.00%] 10.00%] —15.00%| =| 
SigOp 3 1131.50 30 325 | __ 1,487 |47 50.00%] 25.00%] 15.00%) 10.00%] =| SCs 
ComOp1 4124.50 20 325 | __ 4.470 |2,10 90.00%] | 
ComOp2__| 1898.00 20] 219 | 2.437 [2.10 90.00%] | 
ComOp3 1131.50 20 184 | 1,336 |s,10 i 10.00% 
ComOp4 1131.50 20 184 | 1.336 |s,10 90.00% 
CCOP A Aggregated Time to Learn = 3,443 Assumptions: 
CCOP B Time to Learn = 936 (CCOP System Time to Learn is divided evenly over subprocesses in which they operate) 
CCOP C Time to Learn = 594 
CCOP D Time to Learn = 1,825 
CCOP E Time to Learn = 851 
CCOP F Time to Learn = 570 
Tot tur | Tot t; for 1 
Process 
Output 
‘Sub-Process Name ssi (days) 
Pt Review Request/Tasking A 25.00% i 843.70 
P2 Determine Op/Equip Mix A 10 |580 590 492] 10.00%: 550.91 1,082.34 
P3 Input Search Function/Coverage Plan A 35 {1116 1151 492| 20.00% 721.97 1,642.42 
P4 Search/Collection Process A 35 3372 3407 492} 35.00%] 1,684.34 3,898.94 
P5 Target Data Acquisition/Capture A 16 [2245 2261 492] 35.00% 887.49 | 2,356.97 
PBA Signal Type 1 B 342] 35.00%] 410.91 410.91 
P52 Signal Type 2 c 198] 36.00%| 296.91 296.91 
P53 Signal Type 3 D 608] 36.00% 707.24 707.24 
P54 Signal Type 4 E 284| 35.00% 382.57 382.57 
PG Target Data Processing 3401106 1446 723 50.00% 
P64 Signal Type 1 B 312| 50.00% 492.78 
P6.2 Signal Type 2 Cc 198] 50.00% 378.78 378.78 
P63 Signal Type 3 D 608] 50.00% 789.11 789.11 
Signal Type 4 E 284] 50.00%] 464.44 464.44 
PT Target Data Analysis 50 |1698 1748 874 50.00% 
PLA Signal Type 1 B 312} 50.00% §30.52 530.52 
P7.2 Signal Type 2 c 198} 50.00%] 416.62 416.52 
P73 Signal Type 3 D 608] 50.00% 826.85 826.85 
Signal Type 4 E 284| 50.00% 502.18 502.18 
P8 Format Data for Report Generation AF 10 6680 6690 4014 682] 40.00%) 3,357.78 7,371.66 
Pg QC Report AF 30 [848 878 790 682| 10.00% 769.67 | _ 1,560.00 
P10 Transmit Report EF 14_——([1597 1611 242 190} 85.00%] 1,559.28] 1,800.92 
560 12043) 16,310.05 | 26,755.74 
bproce: ASSUMPTIONS 
Review Request/Tasking Days 150.00 
Determine Op/Equip Mix | 554] 534] ~———_—‘1,082 |Avg # Reports during sample period 102 KL Mult 3.00 
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QC Report 
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[722t 920 1.642 [Length of sample period as % 
| 1684] 2.215] 3,899 |Avg # Reports executed/sample p 


| 4559} 421,801 | 
| 16,310[ 12,043] 26,756 | 
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Tota PSK 36877429 Total PBK 24616159 Total P7 K 26920085 Total PBK _649555.09 Total PBK 15911971 TolalPIOK 17753207 
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Market Comparable Price Per Unit (avg) 
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Please note that the floor for ROKA is -100% (e.g., zero return on knowledge assets) |Please note that the floor for ROKA is -100% (e.g., zero return on knowledge assets) 
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KVA Metrics for CCOP C K 






































Sub-Process 
ame non 
Receive/ 
Review 
Request! 
PI Tasking 
Determine 
P2 Op/Equip Mix 
Load Search 
Func/ Coverage 
P3 Plan 
Search/ 
PS Collection 
Target Data 
Acquisition/Cap 
P5 ture 0.00 0.32%| -31086.78%| _-99.68% 
Target Data 
P6 Processing 0.00 0.41%] -24345.97%| _-99.59%| 
Target Data 
PT Analysis 0.00 0.45%] -22130.96%| _-99.55%. 
Format Data for 
Report 
PB Generation 
P9 QC Report 
P10 Transmit Report 
Metrics for Aggregated 0.04 1.18%] -77563.71%} —_-298.82%| 
KVA Metrics for CCOP E K 
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PI Tasking 
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Target Data 
Acquisition/Ca 

P5 |pture 22.32% 
Target Data 

P6 Processing 27.09% 
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PT Analysis 29.29% 
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‘A Metrics for CCOP D K 
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145.99%| 31.50% 
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195.18%| 




















Div LPO 
SigOp 1 
SigOp2 | 1898. 00] 





PERSONNEL TIME SPENT PER PROCESS 


iad a a 36.00% | 


a 
10.00%] 20. 20.00% i 5. 2s —s] 
20.00%| 30.00%| 20.00% 10.00% 


50.00%| 25.00%] 10.00% 

















25.00%) 15.00% 





SigOp 3 r 1131.50] 
ComOp1 4124.50] 
ComOp2__| 1131.50] 
ComOp3 



















CCOP A Aggregated Time to Learn = 
CCOP B Time to Learn = 
CCOP C Time to Learn = 
CCOP D Time to Learn = 
CCOP E Time to Learn = 
CCOP F Time to Learn = 














TF 90. 009 FT 10.00% 


Assumptions: 
(CCOP System TTL is divided evenly over subprocesses in which it operates) 



























































































































































Sub-Process Name 
PI Review Request/Tasking 
P2 Determine Op/Equip Mix 
P3 Input Search Function/Coverage Plan 
P4 Search/Collection Process 
PS Target Data Acquisition/Capture 
P54 Signal Type 1 B | 8 3 § 
P52 Signal Type 2 Cc 198 281.32 
P53 Signal Type 3 D 608 691.66 
P54 Signal Type 4 e 366.99 
P6 Target Data Processing 340 [928 1268 634) 
P61 Signal Type 1 B 470.51 
P62 Signal Type 2 Cc 356.51 
P63 Signal Type 3 D 766.85 
Signal Type 4 E 442.18 
P7 Target Data Analysis 50 1520 1570 785 
P74 Signal Type 1 B 508.25 
P7.2 Signal Type 2 c 394.25 
P7.3 Signal Type 3 D 804.59 
Signal Type 4 Z 284] 50.00% 479.92 
PS Format Data for Report Generation AF 10 (5166 5176 3106) 682| 40.00%. 152.3 5,858.22 
Pg QC Report AF 30__[670 700 630 682| 10.00% 1,381.88 
P10 Transmit Report F 14__[574 778.04 
560 9779 74,256.34 22,616.34 
bees ty 
Subprocess Name ASSUMPTIONS 
Review Request/Tasking i a TI [SamplePa] PriorPd | Days 170.00 
Determine Op/Equip Mix | 551] ~—————531{ 1,082 |Avg # Reports during sample period 368 KL Mult 3.00 
Input Search Function/Coverage Plan | «651 635] 1,286 | Length of sample period as % 100.00% 0.00% 
Search/Collection Process Avg # Reports executed/sample p 368 
Target Data Acquisition/Capture P25] 1.238] 2,063 | 
1 P95 f 395 | 
2 a en 
3 EE ae ee) 
4 P67 fT 867| 
Target Data Processing a E <) e 
1 EE | 7) 
2 Ee ae a) 
3 EE a ee 
4 ay a ee 7) 
Target Data Analysis ae SSS 
1 [soe] | 00] 
2 a a 
3 a ee 
4 a) ee 
Format Data for Report Generation 
QC Report P52] 30] 1,382 | 
Transmit Report a a) cd 
| 14.256] 9.779] 22.616 | 
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CCOP A 
‘CCOP B 
CCOP C 














= | + i 
183997.31, TotalP3K 21865011 TotalP4K  3494089.26 
| | | | 








= | | | i | | 
74341649 TotalP6K  579765.94 —TotalP7K 665133.82 —TotalP&K 2155823.91 TotalP9K  508520.79 TotalP10K  286318.72 





Total PSK 
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KVA Metrics for CCOP C K KVA Metrics for CCOP D K 
Sub-Process Sub-Process 
Name ROK as Ratio | ROK as % ROKA Name ROK as Ratio] ROK as % ROKA ROKI 
Receive! Receive! 
Review Review 
Request! Request! 
PI Tasking PI Tasking 
Determine Determine 
P2 Op/Equip Mix P2 Op/Equip Mix 
Load Search 
Load Search Func! 
Func/ Coverage Coverage 
P3 Plan P3 Plan 
Search! Search/ 
P4 Collection P4 Collection 
Target Data Target Data 
|Acquisition/Cap Acquisition/Ca 
P5 ture 1.46 146.44% 31.71% P5 pture 1.07 106.84% 6.40% 6.84% 
Target Data Target Data 
PG Processing 1.86 185 58% 46.12% P6 Processing 1.18 118.45% 15.58% 18.45% 
Target Data Target Data 
PT Analysis 2.05 205.23%| 51.27% PT Analysis 1.24 124.28% 19.54% 24.28% 
Format Data for Format Data 
Report for Report 
P8 Generation P8 Generation 
a QC Report Pg QC Report 
P10 _ Transmit Report ae Pia 
Metrics for 5.37 537.25%!| 129.10% Metrics for A i 350 35 57% Tsa% W5T% 
KVA Metrics for CCOP E K KVA Metrics for CCOP F K 
Process Sub-Process 
nova 
Receive/ i 
Review 
Request/ 
PI Tasking PI 
Determine 
P2 |Op/Equip Mix 
Load Search 
Func! 
Coverage 
P3 Plan P3 
Search! 
P4 Collection P4 
Target Data 
Acquisition/Ca 
PS pture P5 
Target Data 
PG Processing P6 
Target Data 
PT Analysis 4.15 414.91% 75.90% 314.91% P7 
Format Data 
for Report 
PB Generation Ps 715.92% 
P9 QC Report Pg 122.88%| 
Transmit 
P10 Report P10 308.99% 
Metrics for Aggregated 11.14 a 114.47%| 218.22%| 814. 47% Metrics for Aggregated 14.48 1447.79%| 218.43%| —1147.79%| 
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